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Program description 


Positive Action, a K-12 program, aims to promote character 
development, academic achievement, and social-emotional 
skills and to reduce disruptive and problem behavior. The 
program is based on the philosophy that you feel good about 
yourself when you think and do positive actions, and there is 
always a positive way to do everything. The curriculum includes 
six units; some grades have a review for a seventh unit. All 


lessons are scripted and use classroom discussion, role-play, 
games, songs, and activity sheets or text booklets. Optional 
components that may or may not be implemented as part of the 
program are: site-wide climate development; drug education for 
grade 5 and middle school; conflict resolution; counselor, par- 
ent, and family classes; and community/coalition components. 



RBSearch One study of Positive Action met the What Works Clearinghouse 
(WWC) evidence standards, and one study met standards with 
reservations. The studies included 56 elementary schools In 
Florida and Hawaii. Both studies examined results on students’ 
behavior and academic achievement.'' 



The WWC considers the extent of evidence for Positive Action 
to be moderate to large for behavior and for academic achieve- 
ment. No studies of Positive Action that met WWC evidence 
standards with or without reservations addressed outcomes in 
the domain of knowledge, attitudes, and values. 



EffBCtiVBneSS Positive Action was found to have positive effects on elementary school students’ behavior and academic achievement. 



Knowledge, attitudes, 

Behavior and values Academic achievement 

Rating of effectiveness Positive effects na Positive effects 

Improvement index^ Average: +19 percentile points na Average: +14 percentile points 

Range: -12 to +36 percentile points Range: +8 to +36 percentile points 



na = not applicable 



1. The evidence presented in this report is based on availabie research. Findings and conclusions may change as new research becomes avaiiable. 

2. These numbers show the average and range of improvement indices for aii findings across the two studies and the availabie student-ievei improvement 
indices. 
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Developer and contact 

Positive Action, Inc. 264 4th Avenue South, Twin Falls, ID 83301. 
Web: http://www.positiveaction.net . E-maii: info@positiveaction. 
net. Telephone: (800) 345-2974. Fax: (208) 733-1590. 

Scope of use 

The Positive Action program was created and first implemented 
in four school sites in Idaho from 1979-81. Positive Action, Inc. 
was founded in 1982. Since then, more than 11,000 schoois in 
2,500 schooi districts and 2,000 community groups and agencies 
in ali 50 states have adopted the program. Positive Action may 
have changed since the studies were conducted. The WWC 
recommends asking the developer for information about the most 
current version of this curriculum and taking into account that 
student demographics and school context may affect outcomes. 

Teaching 

The Positive Action curriculum for each of the elementary school 
grades averages 140 lessons taught four days a week, organized 
into six teaching units and a seventh review unit, presented in 
a consistent order. Units 1-3 and a summary of units 4-7 are 
taught in grade 7, totaling 81 lessons, while units 4-7 and a 
summary of units 1-3 are taught in grade 8, totaling 75 lessons. 
The high school curriculum is composed of four kits. Each kit 
covers one school year but can stand on its own or be used in a 
different order. Each kit includes six teaching units and a review 
unit and has approximately 132 lessons. Kit IV, however, has two 
options, each with its own curriculum, so the possible number of 
lessons doubles for that kit. In each of the elementary, middle, or 
high school grade levels teachers may implement the entire set 
of lessons or choose a partial implementation of at least 20 les- 
sons per grade level. There are Leveis of Implementation Plans 
for the Bronze, Silver, Gold, and Platinum Levels, which guide 



Thirteen studies reviewed by the WWC investigated the effects 
of the Positive Action program. One study of the impact of Posi- 
tive Action on elementary school students met WWC evidence 
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users on how to choose lessons for the highest level of effective- 
ness. In all grade levels atypical lesson lasts about 15 minutes. 

The Positive Action program consists of a core curriculum 
and optional components that may or may not be implemented 
as part of the program— site-wide climate development; drug 
education for grade 5 and middle school; conflict resolution; 
counselor, parent, and family classes; and community/coalition 
development. The program philosophy, “You feel good about 
yourself when you think and do positive actions, and there is 
a positive way to do everything,” underlies the content of the 
program units. Kits for the core curriculum and the supplemental 
components may include instructor’s manuals, group materials, 
activity booklets or activity sheets, text booklets, posters, song 
CDs with lyrics, game boards, and teaching aides (stickers, 
index cards, medals, puppets, flannel boards and figures, bean 
bags, and visual aids). 

Cost 

The cost of a Positive Action classroom kit varies by grade level. 
Kindergarten kits cost $460; grades 1-8 cost $360; high school 
kits I, II, and III cost $360; and high school kit IV costs $460. 

The cost of the drug education supplemental kits also varies by 
grade level, ranging from $250 to $360. The optional community 
kit costs $550, the counselor kit $125, and the family kit $75. 
Family classes cost $360 and parenting classes $160. Some of 
the kits are available in Spanish. Combo kits (at reduced prices), 
refresher kits, and kit parts are also available for varied prices. 

An orientation training workshop that includes curriculum; 
climate development; and counselor, family, and community 
components costs $200 a school type (elementary, middle, or 
high school). Ongoing training workshops and media training 
workshops cost $250 each. Workshops that combine orientation, 
ongoing training, and media training for grades K-12 cost $900. 



standards, and one study met standards with reservations. One 
study (Flay, Acock, Vuchinich, & Beets, 2006) was a randomized 
controlled trial. The second study (Flay & Allred, 2003) used a 
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quasi-experimental design. The remaining 10 studies did not 
meet WWC evidence screens. 

Flay and Allred (2003) included 36 elementary schools in 
a large Southeastern school district. The study compared 
outcomes for students participating in the Positive Action 
program with the outcomes for students in schools that did 
not use this program. The study focused on Positive Action as 
implemented schoolwide. 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or moderate to large (see the What Works Clearinghouse 
Extent of Evidence Categorization Scheme) . The extent of 
evidence takes into account the number of studies and the 
total sample size across the studies that met WWC evidence 
standards with or without reservations.^ 

The WWC considers the extent of evidence for Positive Action 
to be moderate to large for behavior and for academic achieve- 
ment. 



Findings 

The WWC review of character education addresses student 
outcomes in three domains: behavior; knowledge, attitudes, and 
values: and academic achievement. 

Behavior. Flay and colleagues (2006) reported statistically sig- 
nificant differences favoring the intervention groups on students’ 
suspension rates, use of alcohol, being drunk, and use of tobacco 
and illegal drugs. The study also reported statistically significant 
differences favoring the intervention group on serious violence 
among boys but not among girls. The average effect size across 
all behavior outcomes in this study was statistically significant."* 

Flay and Allred (2003) reported statistically significant dif- 
ferences favoring the intervention group on students’ violent 
behavior and suspension rates. In addition, the average effect 
size was statistically significant. 

Academic achievement. Flay and colleagues (2006) reported, 
and the WWC confirmed, statistically significant differences 
favoring the intervention groups on students’ grade retention 
rates. The study also examined the impact on state standard- 



ized test scores in reading and math but found no statistically 
significant differences. In addition, the average effect size was 
statistically significant. 

Flay and Allred (2003) reported, and the WWC confirmed, 
statistically significant differences favoring the intervention 
group on total scores of the Florida Comprehensive Aptitude 
Test (FCAT). The authors reported no statistically significant 
impacts on absenteeism. The average effect size across the two 
outcomes in this domain was statistically significant. 

Rating of effectiveness 

The WWC rates interventions as positive, potentially positive, 
mixed, no discernible effects, potentially negative, or negative. 
The rating of effectiveness takes into account four factors: the 
quality of the research design, the statistical significance of the 
findings (as calculated by the WWC), the size of the difference 
between participants in the intervention condition and the com- 
parison condition, and the consistency in findings across studies 
(see the WWC Intervention Rating Scheme) . 



3. The Extent of Evidence categorization was deveioped to teil readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additionai factors associated with a reiated concept, externai vaiidity, such as students’ demographics and the types of set- 
tings in which studies took piace, are not taken into account for the categorization. 

4. The ievei of statisticai significance was reported by the study authors or, where necessary, caicuiated by the WWC to correct for ciustering within ciass- 
rooms or schoois and for muitipie comparisons. For an expianation, see the WWC Tutorial on Mismatch . See the Technical Detaiis of WWC-Conducted 
Computations for the formuias the WWC used to caicuiate the statisticai significance. In the case of Positive Action, corrections for both ciustering and 
muitipie comparisons were needed. 
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The WWC found Positive 
Action to have positive 
effects on both behavior and 
academic achievement of 
eiementary schooi students 



Improvement index 

For each outcome domain, the WWC computed an improvement 
index based on the effect size (see the Technical Details of 
WWC Conducted Computations) . The improvement index repre- 
sents the difference between the percentile rank of the average 
student in the intervention condition versus the percentile rank 
of the average student in the comparison condition. Unlike the 
rating of effectiveness, the improvement index is entirely based 
on the size of the effect, regardless of the statistical significance 
of the effect, the study design, or the analysis. The improvement 
index can take on values between -50 and +50, with positive 
numbers denoting favorable results. The improvement index 
for behavior is +19 percentile points, with a range of -12 to 
+36 percentile points. The improvement index for academic 



achievement is +14 percentile points, with a range of +8 to +36 
percentile points. 

Summary 

The WWC reviewed 13 studies on Positive Action. One study met 
WWC evidence standards, and one study met standards with 
reservations. Both studies assessed elementary school student 
outcomes in the behavior and academic achievement domains. 
When the WWC aggregated the results in each of these domains, 
the average effect sizes were statistically significant. So the WWC 
rated the program as having positive effects on both behavior and 
academic achievement. Character education, an evolving field, is 
beginning to establish a research base. The evidence presented in 
this report is limited and may change as new research emerges. 
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Appendix 



Appendix A1.1 Study characteristics: Fiay, Acock, Vuchinich, & Beets, 2006 (randomized controiied triai) 



Characteristic 


Description 


Study citation 


Flay, B., Acock, A., Vuchinich, S., & Beets, M. (2006). Progress report of the randomized trial of Positive Action in Flawaii: End of third year of infervention. Available from 
Positive Action, Inc. 264 4th Avenue South, Twin Falls, ID 83301. 


Participants 


The study tracked students who started grades K-1 in the 2001-02 academic year. The report reviewed by the WWC examined the outcomes of 2,666 third- and fourth- 
grade students at the end of the third year of program implementation. These outcomes included students who had remained in the school since the beginning of the study 
as well as new students in the school. About 25% of the students surveyed each year were new to the school. The students were enrolled in 20 elementary schools that were 
randomly assigned into conditions. A small percentage of the students were white (13.40% in the intervention school and 17.94% in the comparison). Among the remaining 
students the following ethnic groups were represented: Flawaiian, Filipino, Asian (other), Japanese, Samoan, Flispanic, Chinese, Black, and Portuguese. About 60% of the 
schools in the sample were Title 1 schools. 


Setting 


The study was conducted in 20 K-5 or K-6 schools in Flawaii. Five pairs of matched participating schools were from Oahu, three pairs were from Maui, and two pairs were 
from Molokai. 


Intervention 


The program consisted of the Positive Action curriculum and additional components that involved school principals, counselors, parents, and community members. The 
components addressed school and classroom management, school climate, family and community involvement, and skills and knowledge related to core values.^ The report 
reviewed by the WWC presented findings for the end of the third year of program implementation. The authors report that by the third year, two schools were still implementing 
at a very low level, three at a moderate-to-high level, and five at a high level. But even the high-implementation schools were still not implementing at the level the program 
developer would expect for high-implementation schools. For example, few schools implemented the family- or community-involvement programs. 


Comparison 


The comparison schools were drawn from the same school districts as the intervention schools and were matched on demographic characteristics, student behavior, and 
academic achievement. Comparison schools did not implement the Positive Action program. The comparison schools had other types of character education activities they 
regarded as business-as-usual, which were also practiced (although to a lesser extent) in the intervention schools.^ 


Primary outcomes 
and measurement 


The study examined students’ outcomes in the academic achievement and behavior domains. ^ Outcomes in the academic achievement domain included percent proficient on 
the reading and math sections of a state standardized test and daily absences. Outcomes in the behavior domain included students’ reports of use of alcohol, tobacco, and 
illegal drugs. (See Appendices A2.1 and A2.2 for more detailed descriptions of outcome measures.) 


Teacher training 


Prior to each school year, the program developer, Dr. Allred, provided teacher/staff training to each intervention school. The training lasted 3-4 hours the first year and 1-2 
hours each of the subsequent years. In addition. Dr. Allred visited each school at least once each year to provide an in-service session (usually 30-50 minutes). Finally, 5-6 
representatives from each intervention school participated each winter in a mini-conference to obtain further training on the schoolwide components of the program and to 
share experiences. 



1. Schools in the intervention and comparison groups implemented additional character and behavior programs at the time of the study, which they regarded as business-as-usual. These programs include the Hawai’i State 
Department of Education (HIDOE) mandated General Learner Outcomes (GLO), TRIBES, Lions Quest, and additional programs related to school management and organization and drug and violence prevention, such 

as Positive Behavioral Support, D.A.R.E., Red Ribbon Week, Peace Builders, and Peace Week. The study authors informed the WWC that these schools had some of the additional programs in place before the time of 
pretest. Because the same practices took place in both study conditions, and because greater time dedicated to those additional practices was reported for the comparison group than for the intervention group, this study 
was reviewed. The WWC cautions that interaction effects between Positive Action and the additional programs may have enhanced the program effects. 

2. The study also examined students’ attitudes and values and additional teacher-reported outcomes in the behavior domain. But, since complete statistical information was not available, these outcomes were not reviewed. 
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Appendix A1.2 Study Characteristics: Fiay & Aiired, 2003 (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Flay, B. R., & Allred, C. G. (2003). Long-term effects of the Positive Action program. American Journal of Healthy Behavior, 27(1), 6-21. 


Participants 


The study inciuded 36 eiementary schools. About 62% of the students In the Intervention group participated in the free or reduced-price lunch program compared with 67% 
in the comparison group. About haif of the students were white (50.59% in the intervention group and 44.66% in the comparison group). About one-fourth of the students 
were African-American (24.61% in the intervention group and 28.48% in the comparison group). About one-fifth of the students were Hispanic (20.71% in the intervention 
group and 23.23% in the comparison group). Because schoois in the intervention and comparison conditions were matched, there were no statistically significant differences 
in demographic characteristics between the two groups. 


Setting 


The participating schools were from one large Southeastern school district. The school district was characterized by a large number of school sites (about 65) that have 
implemented Positive Action for four or more years. 


Intervention 


The program consisted of the Positive Action curriculum and additional activities that involved school principals, parents, and community members. The components 
addressed school and classroom management, learning climate, and skills and knowledge related to core values.^ 


Comparison 


The comparison schools were drawn from the same school district as the intervention schools and were matched on demographic characteristics. Comparison schools did not 
implement the Positive Action program. 


Primary outcomes 
and measurement^ 


The study measures in the behavior domain included violence and suspensions rates. The study measures in the academic achievement domain included the Florida Compre- 
hensive Aptitude Test (FCAT) and grade retention rates. (See Appendices A2.1 and A2.2 for more detailed descriptions of outcome measures.) 


Teacher training 


No information on teacher training was provided. 



1 . Half of fhe schools In fhe Inferventlon group Implemenfed addlflonal character and behavior programs at the time of the study. The study authors Informed the WWC that these schools had the additional programs In place before the time 
of pretest. That Is, the Positive Action program was the only new addition between the pretest and the posttest. The WWC cautions that Interaction effects between Positive Action and the additional programs may have enhanced the 
program effects. 

2. Flay and Allred (2003) also Investigated the long-term Impact of Positive Action on students In middle and high schools. The study conducted school-level analyses comparing three categories of schools: less than 60%, between 60% and 
79%, and between 80% and 100% of the student population with exposure to the elementary school Positive Action program. But, because those analyses did not use a comparison group, they were not reviewed In this WWC report. 



WWC Intervention Report Positive Action 



Revised April 23, 2007 




Appendix A2.1 Outcome measures in the behavior domain 



Outcome measure 


Description 


Violence rates 


School-level archival data that consist of disciplinary referrals for incidents of violence per 100 studenfs (as cited in Flay & Allred, 2003). 


Suspensions rates 


School-level archival data that consist of percent of students who received out-of-school suspensions (as cited in Flay & Allred, 2003). 


Tobacco use 


One item rated on a three-point scale on which students indicate if they have ever smoked cigarettes (as cited in Flay et al., 2006). 


Alcohol use 


One item rated on a three-point scale on which students indicate if they have ever drunk alcoholic beverages (as cited in Flay et al., 2006). 


Being drunk 


One item rated on a three-point scale on which students indicate if they have ever been drunk (as cited in Flay et al., 2006). 


Illegal drug use 


One item rated on a three-point scale on which students indicate if they have ever used illegal drugs (as cited in Flay et al., 2006). 


Violent behavior 


A self-report survey that measures level of engagement in five serious behaviors: carried a knife or razor to use to hurt someone, threatened to cut or stab someone, cut or 
stabbed someone on purpose to hurt them, carried a gun, and shot at someone (as cited in Flay et al., 2006). 


Suspensions 


School-level archival data that consist of the rates at which students were suspended during the school year (as cited in Flay et al., 2006). 



Appendix A2.2 Outcome measures in the academic achievement domain 



Outcome measure 


Description 


The Florida Comprehensive 
Aptitude Test (FCAT) 


This standardized test contains two basic components: criterion-referenced tests (CRT), measuring selected benchmarks in mathematics, reading, science, and writing 
from the Sunshine State Standards (SSS); and norm-referenced tests (NRT) in reading and mathematics, measuring individual student performance against national norms. 
Total score for the grade 4 FCAT was used in the reviewed study (as cited in Flay & Allred, 2003). 


The Hawaii Content and 
Performance Standards 
test (HCPS) 


A state standardized academic achievement test. Fifth-grade test scores on the reading and math subtests were used in this study (as cited in Flay et al., 2006). 


Absentee rates 


School-level archival data that consist of percent of students absent for 21 or more days during the school year (as cited in Flay & Allred, 2003). 


Retention in grade 


School-level archival data that consist of the number of students retained in the same grade level (as cited in Flay et al., 2006). 
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Appendix A3.1 Summary of study findings inciuded in the rating for the behavior domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 








Sample size 




Mean difference^ 


Statistical 






Study 


(students/ 


Positive Action Comparison 


(Positive Action 


significance^ 


Improvement 


Outcome measure 


sample^ 


schools) 


group group 


- comparison) 


Effect size^ (at a = 0.05) 


index^ 









Flay et al., 2006 (randomized controlled trial) 










Suspensions 


Grade 5 


2,660/20 


1.63 to 1,000 
students 


6.2 to 1,000 
students 


0.01 


0.81 


Statistically 

significant 


+29 


Tobacco use 


Grade 5 


2,660/20 


3.33 (na) 


4.78 (na) 


1.45 


0.23 


Statistically 

significant 


+9 


Alcohol use 


Grade 5 


2,660/20 


9.63 (na) 


14.83 (na) 


5.20 


0.30 


Statistically 

significant 


+12 


Being drunk 


Grade 5 


2,660/20 


0.74 (na) 


3.35 (na) 


2.61 


0.93 


Statistically 

significant 


+32 


Illegal drug use 


Grade 5 


2,660/20 


0 (na) 


2.39 (na) 


2.39 


1.08 


Statistically 

significant 


+36 


Serious violence (boys) 


Grade 5 


1,330/20 


10.8 (na) 


23.5 (na) 


12.70 


0.56 


Statistically 

significant 


+21 


Serious violence (girls) 


Grade 5 


1,330/20 


5.8 (na) 


3.6 (na) 


-2.20 


-0.30 


ns 


-12 


Average^ for behavior (Flay et al., 2006) 










0.52 


Statistically 

significant 


+20 








Flay & Allred, 2003 (quasi-experimental design) 








Violence rates 


Grades 1-6 


36 schools 


3.83 (na) 


12.11 (na) 


8.28 


0.75 


Statistically 

significant 


+27 


Suspensions rates 


Grades 1-6 


36 schools 


2.72 (na) 


4.09 (na) 


1.37 


0.25 


Statistically 

significant 


+10 


Average^ for behavior (Flay & Allred, 2003) 










0.50 


Statistically 

significant 


+19 


Domain average^ across studies 












0.51 


na 


+19 



ns = not statistically significant 
na = not applicable 



1 . This appendix reports findings considered for the effectiveness rating and the improvement index. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

(continued) 
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Appendix A3.1 Summary of study findings inciuded in the rating for the behavior domain^ (continued) 



3. Flay and Allred (2003) also Investigated the Impact of Positive Action on secondary schools, but this study did not meet WWC evidence screens because It did not use a comparison group. 

4. Positive differences and effect sizes favor tbe Intervention group; negative differences and effect sizes favor the comparison group. The sign for the direction of effect for outcomes Indicating problem behavior was reversed, to Indicate a 
positive effect that Is associated with a small rate of problem behavior In the Intervention schools. 

5. Effect sizes for all outcomes In the behavior domain were calculated using the odds ratio formula. For an explanation of the effect size calculation, please see the Technical Details of WWC-Conducted Computations . 

6. Statistical significance Is the probability that the difference between groups Is a result of chance rather than a real difference between the groups. The level of statistical significance was reported by the study authors or, where neces- 
sary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See the Technical Details of WWC- 
Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Positive Action, corrections for both clustering and multiple comparisons were needed. 

7. The Improvement Index represents the difference between the percentile rank of the average student In the Intervention condition and that of the average student In the comparison condition. The Improvement Index can take on values 
between -50 and -i-50, with positive numbers denoting favorable results. 

8. The WWC-computed average effect sizes are simple averages rounded to two decimal places. The average Improvement Indices are calculated from the average effect sizes. 
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Appendix A3.2 Summary of study findings inciuded in the rating for the academic achievement domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 








Sample size 




Mean difference^ 


Statistical 






Study 


(students/ 


Positive Action Comparison 


(Positive Action 


significance^ 


Improvement 


Outcome measure 


sample^ 


schools) 


group group 


- comparison) 


Effect size^ (at a = 0.05) 


index^ 



The Hawaii Content and 
Performance Standards test 
(HCPS)— math (percentage 
reaching proficiency) 


Grade 5 


20/2,660 


Flay et al., 2006 (randomized controlled trial) 

26.00 (na) 21.00(na) 5.00 


0.17 


ns 


+7 


The Hawaii Content and 
Performance Standards test 
(HCPS) — reading (percentage 
reaching proficiency) 


Grade 5 


20/2,660 


52.00 (na) 


44.00 (na) 


8.00 


0.19 


ns 


+8 


Grade retention 


Grade 5 


20/2,660 


1 to 1,000 students 


6 to 1 ,000 students 


0.01 


1.09 


Statistically 

significant 


+36 


Average^ for academic achievement (Flay et al., 2006) 








0.48 


Statistically 

significant 


+19 








Flay & Allred, 2003 (quasi-experimental design) 








The Fiorida Comprehensive 


Grade 4 


36 schoois 


290.90 (nr) 


278.40 (nr) 


12.50 


na® 


Statisticaiiy 


na® 


Aptitude Test (FCAT) 














significant 




Absentee rates 


Grades 1-6 


36 schoois 


10.79 (na) 


12.36 (na) 


1.57 


0.25 


ns 


+10 


Average^ for academic achievement (Flay & Allred, 2003) 








0.25 


Statistically 


+10 
















significant 




Domain average^ across studies 










0.37 


na 


+14 



ns = not statistically significant 
nr = not reported 
na = not applicable 

1 . This appendix reports findings considered for the effectiveness rating and the improvement index. One additional finding based on a subfest of the Florida Comprehensive Aptitude Test (FCAT) was not used for rafing purposes because 
of an overlap with total FCAT score, but is presented in Appendix A4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. Flay and Allred (2003) also investigated the impact of Positive Action on secondary schools, but this study did not meet WWC evidence screens because it did not use a comparison group. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The sign for the direction of effect for absentee rates was reversed, to indicate a positive effect that is 
associated with smaller absentee rates in the intervention schools. 

(continued) 
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Appendix A3.2 Summary of study findings inciuded in the rating for the academic achievement domain^ (continued) 



5. Effect sizes for all outcomes reviewed for Flay and Allred (2003) and Flay et al. (2006), except for students scores on the FCAT in Flay and Allred (2003), were calculated using the odds ratio formula. For an explanation of the effect size 
calculation, please see the Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The level of statistical significance was reported by the study authors or, where neces- 
sary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See the Technical Details of WWC- 
Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Positive Action, corrections for both clustering and multiple comparisons were needed. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and -t-50, with positive numbers denoting favorable results. 

8. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. 

9. Student-level standard deviations were not available in Flay and Allred (2003) for the FCAT results. The study provided school-level pooled standard deviations but no standard devations by condition. Because the student-level effect 
size and improvement index could not be computed, the magnitude of the effect size for this outcome measure was not considered for rating purposes. Flowever, the statistical significance for this outcome is comparable to other studies 
and is included in the intervention rating. For further details, please see Technical Details of WWC-Conducted Computations . 
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Appendix A4 Summary of additional study findings for the academic achievement domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 








Sample size 




Mean difference^ 


Statistical 






Study 


(students/ 


Positive Action Comparison 


(Positive Action 


significance^ 


Improvement 


Outcome measure 


sample 


schools) 


group group 


- comparison) 


Effect size^ (at a = 0.05) 


index^ 



Florida Reading Test 


Grade 4 


36 schools 


Flay & Allred, 2003 (quasi-experimental design) 

105.90 (nr) 73.10 (nr) 32.80 


1.29 


Statistically 


-f40 












significant 





nr = not reported 



1 . This appendix presents an additionai finding that faiis in the academic achievement domain. This is a subtest of the Fiorida Comprehensive Aptitude Test (FCAT). Findings for totai score on the FCAT are presented in Appendix A3. 2. 

2. The standard deviation across aii students in each group shows how dispersed the participants’ outcomes are: a smaiier standard deviation on a given measure wouid indicate that participants had more simiiar outcomes. For the caicu- 
iation of an effect size pooied standard deviations, which were converted to student-ievei standard deviations, were used. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an expianation of the effect size caicuiation, piease see the Technicai Detaiis of WWC-Conducted Computations . 

5. Statisticai significance is the probabiiity that the difference between groups is a resuit of chance rather than a reai difference between the groups. The ievei of statisticai significance was reported by the study authors or, where neces- 
sary, caicuiated by the WWC to correct for ciustering within ciassrooms or schoois and for muitipie comparisons. For an expianation about the ciustering correction, see the WWC Tutoriai on Mismatch . See the Technical Detaiis of WWC- 
Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Positive Action, a correction for clustering was needed. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and -t-50, with positive numbers denoting favorable results. 
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Appendix A5.1 Positive Action rating for the behavior domain 



The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negativeJ 
For the outcome domain of behavior, the WWC rated Positive Action as having positive effects. The remaining ratings (potentially positive effects, mixed effects, no 
discernibie effects, potentially negative effects, and negative effects) were not considered, because Positive Action was assigned the highest applicabie rating. 



Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaily significant positive effects, at ieast one of which met WWC evidence standards for a strong design. 

Met. Positive Action had two studies, one of which met WWC evidence standards for a strong design. Both studies showed statistically significant 
positive effects. 

• Criterion 2: No studies showing statistically significant or substantiveiy important negative effects. 

Met. The WWC anaiysis found no statisticaiiy significant or substantively important negative effects in this domain. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive effects. See the WWC 
Intervention Rating Scheme for a complete description. 
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Appendix A5.2 Positive Action rating for the academic achievement domain 



The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negativeJ 
For the outcome domain of academic achievement, the WWC rated Positive Action as having positive effects. The remaining ratings (potentially positive effects, 
mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered, because Positive Action was assigned the highest appli- 
cable rating. 

Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Positive Action had two studies, one of which met WWC evidence standards for a strong design. Both studies showed statistically significant 
positive effects. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive effects. See the WWC 
Intervention Rating Scheme for a complete description. 
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Appendix A6 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schools 


Sample size 

Students 


Extent of evidence^ 


Behavior 


2 


56 


over 2,660 


Moderate to large 


Knowledge, attitudes, and values 


0 


0 


0 


na 


Academic achievement 


2 


56 


over 2,660 


Moderate to large 



na = not applicable/not studied 

1. A rating of “moderate to large” requires at least two studies and two schools across studies in one domain, and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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